Skip to content

fix(flink): improve read path coverage and CDC reliability - #19402

Merged
danny0405 merged 3 commits into
apache:masterfrom
cshuo:flink-read-path-coverage
Jul 30, 2026
Merged

fix(flink): improve read path coverage and CDC reliability#19402
danny0405 merged 3 commits into
apache:masterfrom
cshuo:flink-read-path-coverage

Conversation

@cshuo

@cshuo cshuo commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Describe the issue this Pull Request addresses

Closes #19401.

The Flink table-format and source-reader read paths lacked focused coverage for partition paths, CDC iteration, schema reconciliation, COW split planning, Lance reads, and source split-reader lifecycle behavior. The added CDC coverage also exposed two correctness gaps: an empty CDC file split could stop iteration before later changes, and truncated image bytes could cause an unbounded skip loop.

Summary and Changelog

  • Add and extend focused tests for table-format and source-reader paths, including partition parsing and pruning, schema reconciliation, COW split planning, CDC image caching and iteration, source enumeration, expression conversion, split-reader lifecycle, projections, exception paths, and resource cleanup.
  • Add TestHoodieRowDataLanceReader for metadata, row-key filtering, Arrow batch iteration, schema validation, and resource cleanup.
  • Make CdcFileSplitsIterator advance through empty child iterators until it finds a row or exhausts all file splits.
  • Make BytesArrayInputView throw EOFException when truncated input prevents a requested skip from making progress.

Targeted local JaCoCo line coverage:

Class Before After
FilePathUtils 33% 94.5%
CdcIterators 67% 88.0%
CdcImageManager 64% 91.3%
CopyOnWriteInputFormat 45% 91.2%
HoodieRowDataLanceReader 47% 81.9%
InternalSchemaManager 18% 82.9%
RecordIterators 56% 93.0%
HoodieCdcSplitReaderFunction 16% 80.6%
HoodieSplitReaderFunction 30% 100.0%
ExpressionEvaluators 69% 88.2%
HoodieSource 54% 90.0%
IncrementalInputSplits 81% 87.8%

The JaCoCo-enabled reactor run passed 212 tests with no failures, errors, or skips:

mvn -Punit-tests -Djacoco.skip=false \
  -pl hudi-flink-datasource/hudi-flink -am \
  -Dtest=TestFilePathUtils,TestCdcImageManager,TestCdcIterators,TestInternalSchemaManager,TestCopyOnWriteInputFormat,TestRecordIterators,TestHoodieSource,TestIncrementalInputSplits,TestExpressionEvaluators,TestHoodieSplitReaderFunction,TestHoodieCdcSplitReaderFunction,TestHoodieRowDataLanceReader,TestInputFormat \
  -Dsurefire.failIfNoSpecifiedTests=false \
  -DskipITs -DskipSparkTests -DskipScalaTests test

Impact

  • Functional impact: CDC readers no longer stop at empty file splits, and truncated image input fails fast with EOFException instead of hanging.
  • Maintainability: Focused tests document the supported table-format and source-reader branches and their resource ownership.
  • Extensibility: The new Lance reader test fixture makes additional projections, filters, and metadata cases easier to add.

Risk Level

Low. The production changes are limited to CDC iterator boundary handling and an EOF guard. The affected classes and source-reader caller were covered by targeted tests, and the complete 212-test JaCoCo run passed.

Documentation Update

None.

Contributor's checklist

  • Read through contributor's guide
  • Enough context is provided in the sections above
  • Adequate tests were added if applicable

@codecov-commenter

codecov-commenter commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.54%. Comparing base (45dcf1e) to head (71f5f0a).

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19402      +/-   ##
============================================
+ Coverage     75.36%   75.54%   +0.18%     
- Complexity    32552    32659     +107     
============================================
  Files          2574     2574              
  Lines        142994   142995       +1     
  Branches      17529    17530       +1     
============================================
+ Hits         107762   108026     +264     
+ Misses        27182    26929     -253     
+ Partials       8050     8040      -10     
Components Coverage Δ
hudi-common 82.26% <ø> (-0.01%) ⬇️
hudi-client 81.79% <ø> (-0.02%) ⬇️
hudi-flink 84.03% <100.00%> (+1.43%) ⬆️
hudi-spark-datasource 68.34% <ø> (ø)
hudi-utilities 71.18% <ø> (-0.03%) ⬇️
hudi-cli 15.32% <ø> (ø)
hudi-hadoop 63.49% <ø> (+0.01%) ⬆️
hudi-sync 70.67% <ø> (ø)
hudi-io 79.57% <ø> (ø)
hudi-timeline-service 83.54% <ø> (ø)
hudi-cloud 64.00% <ø> (ø)
hudi-kafka-connect 53.20% <ø> (ø)
Flag Coverage Δ
common-and-other-modules 49.21% <100.00%> (+0.32%) ⬆️
flink-integration-tests 48.84% <100.00%> (+0.32%) ⬆️
hadoop-mr-java-client 43.37% <ø> (-0.03%) ⬇️
integration-tests 13.63% <0.00%> (+<0.01%) ⬆️
spark-client-hadoop-common 48.71% <ø> (+<0.01%) ⬆️
spark-java-tests 51.39% <ø> (+<0.01%) ⬆️
spark-scala-tests 46.09% <ø> (+<0.01%) ⬆️
utilities 36.65% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
.../apache/hudi/table/format/cdc/CdcImageManager.java 91.30% <100.00%> (+27.12%) ⬆️
...org/apache/hudi/table/format/cdc/CdcIterators.java 82.54% <100.00%> (+9.71%) ⬆️

... and 26 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cshuo
cshuo marked this pull request as ready for review July 29, 2026 03:43
@github-actions github-actions Bot added the size:XL PR with lines of changes > 1000 label Jul 29, 2026

@danny0405 danny0405 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found two correctness gaps in the new CDC coverage. Both can cause a reader task to hang or silently skip later changes.

@cshuo
cshuo force-pushed the flink-read-path-coverage branch from 8e8c4e5 to c3ab8ff Compare July 29, 2026 06:55
@cshuo cshuo changed the title test(flink): improve read path coverage fix(flink): improve read path coverage and CDC reliability Jul 29, 2026

@cshuo cshuo left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed both CDC correctness comments in c3ab8ff. The JaCoCo-enabled targeted suite passes 212 tests with no failures, errors, or skips.

@cshuo
cshuo force-pushed the flink-read-path-coverage branch from c3ab8ff to 71f5f0a Compare July 30, 2026 01:35
@hudi-bot

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@danny0405
danny0405 merged commit 46cd134 into apache:master Jul 30, 2026
139 of 144 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL PR with lines of changes > 1000

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FLINK] Improve table format and source reader read path test coverage

4 participants